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Abstract 

Many  DoD  weapon  systems  acquisition  programs  are  exceeding  their  original 
estimates  for  total  ownership  costs.  There  are  probably  many  contributing  factors  to 
this  cost  growth,  but  is  Systems  Engineering  (SE)  one  of  them?  How  can  systems 
engineering  processes,  methods,  and  practices  be  improved  to  better  control  total 
ownership  cost  growth  in  DoD  acquisition  programs?  This  paper  discusses  research 
in  developing  an  understanding  of  how  SE  can  be  optimized  for  developing  high 
confidence  estimates  and  better  control  of  acquisition  program  total  ownership  costs 
(TOC).  Although  this  research  is  in  the  very  early  stages,  we  discuss  the 
technical  approach  to  investigating  systems  engineering  methods  and  practices 
related  to  TOC  as  executed  at  one  of  the  Navy’s  major  system  acquisition 
commends  (Naval  Air  Systems  Command-NAVAIR).  We  discuss  very 
preliminary  findings  and  set  the  stage  for  further  research  results. 

Background 

Total  Ownership  Cost  (TOC)  Definitions 

Many  DoD  weapon  systems  acquisition  programs  are  exceeding  their  original 
estimates  for  total  ownership  costs.  There  are  probably  many  contributing  factors  to  this 
cost  growth,  but  is  Systems  Engineering  (SE)  one  of  them?  How  can  systems  engineering 
processes,  methods,  and  practices  be  improved  to  better  control  total  ownership  cost  growth 
in  DoD  acquisition  programs?  This  paper  discusses  research  in  developing  an 
understanding  of  how  SE  can  be  optimized  for  developing  high  confidence  estimates  and 
better  control  of  acquisition  program  total  ownership  costs  (TOC).  Although  this  research 
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is  in  the  very  early  stages,  we  discuss  the  technical  approach  to  investigating  systems 
engineering  methods  and  practices  related  to  TOC  as  executed  at  one  of  the  Navy’s 
major  system  acquisition  commends  (Naval  Air  Systems  Command;  NAVAIR).  We 
discuss  very  preliminary  findings  and  set  the  stage  for  further  research  results. 

Before  proceeding  further,  it  would  be  useful  to  establish  definitions  for  TOC.  In 
general,  TOC  is  made  up  of  four  categories  of  cost  that  are  incurred  during  the  system 
acquisition  lifecycle.  These  are  not  completely  independent  but  overlap  and  are  associated 
with  four  major  phases  of  the  system  lifecycle.  TOC  is  comprised  of  the  following: 

■  Research  and  development  cost  that  extend  from  the  concept  phase  to  the 
technology  development  phase  and  through  to  development  and 
demonstration, 

■  Costs  associated  with  system  production, 

■  Operations  and  support  cost  during  sustainment  phase,  and 

■  Disposal  and  retirement  costs. 

Another  broad  definition  of  TOC  is  that  “TOC  is  comprised  of  costs  to  research, 
develop,  acquire,  own,  operate,  and  dispose  of  weapon  and  support  systems,  other 
equipment  and  real  property,  the  costs  to  recruit,  train,  retain,  separate  and  otherwise 
support  military  and  civilian  personnel,  and  all  other  costs  of  business  operations  of  the 
DoD”  (Gansler,  1998). 

A  more  specific  example  of  how  TOC  elements  can  be  decomposed  can  be  found  in 
Figure  1.  This  figure  is  derived  from  NAVAIR  discussions  and  their  perspective  of  aviation 
weapons  systems  acquisitions.  TOC,  therefore,  includes  many  components  of  cost  that  go 
well  beyond  simply  the  initial  acquisition  of  the  system. 

TOTAL  OWNERSHIP  COST'  DECOMPOSITION 


Total  Ownership  Cost 

Common  Spares/  Support  Arcrtrs,  infrastructure  Cost  for  P  tanning/  Managing/  Operating^  Executing,  Linked 


Indimcl  ftKHJznwftOhn  IfTWOWnflnts 


L/fe  Cycle  Cost 

*  Opera  tiotn  And  Support  ffttisswn  Personrwf.  Untt-fevet  Consumption,  intermediate  Maintenance, 
Do pal  Malnlen jnco,  Contractor  Support,  Support,  Indtrvc  r  Support).  Disposal 

Acquisition  Cost 

■*  RDT&E,  Facility  Construction 


Procurement  Cost 

*  Initial  Spares 


Weapon  Sysfem  Cost 


-  reef?  dju.  Putt  fit  a  u  cm.  Contractor  Services.  Support  Equipment, 
Training  Equipment,  Factory  Training 


Flyaway  Cost 

*  Nan-recurring.  Ancillary  Equipment 


Recurring  Flyaway 

»  rtijpaflemettf.  Hardware.  ECOf 


Figure  1.  Total  Ownership  Cost  Components  at  NAVAIR 
DoD  Acquisition  Total  Ownership  Cost  Concerns 

DoD  systems  are  often  acquired  with  operational  performance  in  mind  during  the 
engineering  phase.  For  warfare  systems,  this  is  entirely  appropriate  as  the  systems  are 
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usually  employing  leading  technology,  operate  in  challenging  environments,  and  their  failure 
can  result  in  potentially  cataclysmic  national  impact.  The  acquisition  of  warfare  systems  has, 
however,  manifested  a  higher  total  ownership  cost  over  the  lifetime  of  the  system  then  was 
either  predicted  or  anticipated  when  the  program  was  originally  made  a  program  of  record. 
Considerations  for  how  a  system  would  operate  beyond  the  acquisition  cycle  has  often 
fallen  to  the  logisticians,  maintainers,  supply  chain  analysts,  and  communities  involved  with 
operations  and  maintenance.  The  total  cost  of  operating  the  systems,  however,  during  these 
phases  can  easily  exceed  50%  of  the  total  cost  of  the  system  from  birth  until  retirement. 

While  design  for  performance  remains  the  key  objective  for  systems  acquisition, 
designing  for  affordability  is  emerging  as  a  key  companion  objective  which  may  require  new 
system  engineering  practices,  trade-off  analyses,  optimization  and  value  assessment,  and 
other  modifications  to  the  system  engineering  methodologies  currently  employed  in  a 
performance-first  focus.  Additionally,  system  engineering,  logistics,  and  program 
organizational  realignments  also  may  be  necessary  to  influence  the  entire  enterprise  to 
ensure  that  design  for  affordability  is  increased  in  stature  as  compared  to  design  for 
performance.  New  methods  of  engineering  emphasis  on  governance  and  analysis  appear  to 
be  needed  as  major  military  acquisitions  continue  to  exceed  total  ownership  cost  objectives 
and  estimates. 

Senior  leaders  in  the  DoD  and  the  Navy  have  started  to  apply  new  emphasis  to  the 
reduction  of  total  ownership  cost  (R  TOC)  of  systems.  This  emphasis  and  reduction  in  total 
ownership  costs  has  resulted  in  pilot  programs  that  examine  logistics,  maintainability,  and 
supply  chain  management  issues  that  are  intended  to  discover  ways  to  improve  readiness 
and  reduce  logistics  footprint  (Wynne,  2003).  This  has  spawned  research  dedicated  solely 
to  reduction  of  TOC  but  often  from  a  management  (vice  engineering)  perspective 
(Boudreau,  2003).  Recently,  the  Chief  of  Naval  Operations  stated  that  total  ownership  costs 
will  become  a  priority  at  beginning  of  program  start:  “I  tell  my  leaders  if  we’re  going  to  talk 
about  a  program  or  policy  we’re  going  to  start  with  the  discussion  of  total  ownership  costs 
before  we  get  on  to  anything  else.  That’s  absolutely  key”  (Roughead,  2010). 

Additionally,  to  increase  the  visibility  and  measurability  of  affordability  of  a  program, 
the  Under  Secretary  of  Defense  has  recently  recommended  that  affordability  be  mandated 
as  a  requirement  in  any  program  (Carter,  2010). 

It  is  well  understood  that  the  determining  factors  of  total  ownership  cost  are 
established  early  in  the  development  of  the  system.  The  design  decisions,  architectures, 
logistics  strategies,  and  operational  concepts  all  are  established  early  and,  in  effect,  “set  in 
concrete”  the  destiny  of  the  overall  lifecycle  cost  of  the  system.  As  shown  in  Figure  2,  some 
chronic  TOC  problems  are  starting  to  show  up  where  early  estimates  of  total  ownership  cost 
have  proven  to  be  inaccurate,  which  brings  unpleasant  surprises  later  in  the  lifecycle  of 
systems  that  have  been  deployed.  This  error  in  estimation  or  the  inability  to  control 
ownership  cost  is  causing  significant  perturbations  to  the  operational  and  sustainment 
(O&S)  budgets  within  the  DoD. 
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Figure  2.  Dimensions  of  Total  Ownership  Cost  Growth  in  the  Acquisition  Cycle 

Note.  This  figure  was  derived  from  Eggenberger  (2010). 


As  shown  in  Figure  3,  these  O&S  costs  can  represent  70-80%  of  the  total  program 
cost.  The  question  remains,  if  the  trajectory  of  total  ownership  costs  is  set  early  in  the  design 
phase  of  a  program  acquisition,  what  can  be  done  during  those  early  phases  to  improve  the 
accuracy  of  the  estimates  and  ultimately  the  control  of  the  cost  later  in  the  lifecycle? 
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Figure  3.  Distribution  of  Key  TOC  Components 

(Eggenberger,  2010) 


NAVAIR  TOC  Research 


Problem  Definition 

The  Naval  Air  Systems  Command  (NAVAIR)  is  a  highly  experienced  and  technical 
system  acquisition  organization  that  acquires  U.S.  Navy's  aircraft  and  supporting  systems. 
They  have  robust  engineering  and  logistic  processes  that  shepherd  the  acquisition  of  new 
systems  into  the  Naval  aviation  enterprise.  NAVAIR  realizes,  as  with  the  other  systems 
commands,  that  the  estimation  and  control  of  TOC  remains  a  challenge,  especially  in  this 
era  of  high-tempo  combat  operations.  An  investigation  is  underway  to  examine  whether  or 
not  specific  processes  can  be  improved  to  increase  TOC  estimation  accuracy  and  control. 
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The  System  Engineering  Development  and  Implementation  Center  (SEDIC)  at 
NAVAIR  has  been  investigating  how  the  earliest  system  engineering  activities  influence 
TOC  estimation  accuracy  and  control  and  whether  or  not  the  system  engineering  activities 
can  be  improved  upon.  The  authors  have  participated  with  the  SEDIC  to  assist  in  the 
development  of  improved  system  engineering  solutions  and  also  to  examine  the  opportunity 
to  develop  new  SE  metrics  and  tools  to  the  larger  system  engineering  community.  Specific 
problems  and  research  questions  that  we’ve  established  are  listed  as  follows: 

■  Problem:  Systems  engineering  (SE)  is  optimized  for  designing  for  best 
system  performance  but  may  not  be  aligned,  prioritized,  or  defined  well  to 
designing  for  affordability  and  TOC  objectives. 

o  Research  Question  1 :  How  does  SE  contribute  to  TOC  estimation, 
reduction,  and  control  objectives  and  activities? 
o  Research  Question  2:  Can  SE  activities  be  improved,  better  defined, 
or  integrated  into  other  TOC  reduction  activities  to  improve  TOC 
estimation  and  control? 

o  Research  Question  3:  Can  TOC  metrics  be  developed  and 

integrated  into  SE  and  program  activities  and  toolsets  to  quantitatively 
develop  TOC  KPPs,  KSAs,  MOEs,  etc.,  and  quantitatively  assess 
program  performance  against  those  metrics? 

This  research  positions  itself  between  the  system  engineering  activities  that  are 
typically  associated  with  designing  for  performance  and  the  system  engineering  activities 
and  logistics  activities  which  are  often  considered  to  be  designing  for  affordability  (see 
Figure  4).  The  research  is  attempting  to  find  best  practices  that  are  successful  in  developing 
accurate  TOC  strategies  as  well  as  assisting  in  identifying  improvements  and  linkages 
among  these  methodologies. 
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Figure  4.  This  Research  Focuses  at  the  Intersection  of  Design  for  Performance 
and  Design  for  Affordability  Methods  and  Practices 

Previous  and  Related  Methodologies 

The  estimation,  measurement,  and  control  of  acquisition  costs  is  not  a  new  topic,  nor 
is  there  a  lack  of  techniques  that  are  intended  to  control  such  costs.  This  research  is  highly 
focused  on  the  early  system  engineering  activities  and  how  they  can  directly  impact  TOC 
estimation  and  control.  We  acknowledge  proven  bodies  of  knowledge  and  methodologies 
and  will  not  reinvestigate  or  replicate  but,  rather,  will  attempt  to  integrate  the  most  applicable 
facets  of  those  methodologies  into  any  SE  methodology  we  may  be  able  to  discover.  In 
particular,  methodologies  associated  with  (1)  value  methodology,  (2)  O&S  cost  engineering, 
(3)  design  to  cost,  and  (4)  cost  as  an  independent  variable  form  a  substantial  foundation 
upon  which  to  build.  Each  of  these  disciplines,  however,  brings  their  own  perspective  to  cost 
estimation  and  control  and  may  be  enriched  by  enhancing  with  system  engineering  activities 
that  are  also  focused  and  similar  areas.  Following  discussions  highlight  how  these  different 
existing  disciplines  are  focused. 

Value  Methodology 

Value  methodology  (VM;  also  Value  Engineering,  VE),  is  a  structured  approach  used 
to  analyze  manufacturing  products  and  processes,  design  and  construction  projects,  and 
business  and  administrative  processes.  VM  helps  achieve  balance  between  required 
functions,  performance,  quality,  safety,  and  scope  with  system  cost.  The  proper  balance 
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results  in  the  maximum  value  for  the  project  where  value  is  often  the  ratio  of  cost-to- 
functionality  (SAVE,  201 1 ). 

Value  methodology  is  often  implemented  through  a  process  consisting  of  a  series  of 
activities,  including: 

■  Mission  and  requirements  definition, 

■  Functional  analysis, 

■  Alternative  synthesis,  and 

■  Evaluation,  trade-off,  and  selection. 

This  methodology  is  mature  and  SE  has  inherited  and  incorporated  many  of  the  VM 
tenants  in  Functional  Analysis  and  Allocation,  Requirements  Engineering,  and  System 
Analysis  processes.  Applicability  to  this  research  is  the  proposition  of  “value”  and  how  that 
assessment  relates  to  the  “value”  metrics  of  TOC. 

Cost  Engineering 

The  Operating  and  Support  (O&S)  Cost  element  structure  is  often  divided  into  six 
major  categories:  (1)  personnel,  (2)  operations,  (3)  maintenance,  (4)  sustainment  support, 
(5)  system  improvements,  and  (6)  indirect  support  (OSD,  2011).  At  NAVAIR,  the  cost 
process  includes  the  following  activities: 

■  Break-Even  Analysis, 

■  Present  Value  Analysis, 

■  Regression  Analysis, 

■  Forecasting, 

■  Sensitivity  Analysis, 

■  Should  Cost  Analysis, 

■  Cost  Modeling, 

■  Financial  Analysis, 

■  Cost  Data  Analysis, 

■  Proposal  Analysis, 

■  Overhead  Analysis, 

■  Rate  Analysis, 

■  Engineering  Cost  Analysis,  and 

■  Learning  Curve  Application.  (NAVAIR,  201 1 ) 

The  varied  analysis  and  modeling  activities  mentioned  previously  are  highly 
dependent  on  accurate,  high-fidelity  engineering  inputs  in  order  to  produce  high-confidence 
cost  estimates.  The  focus  of  this  research  is  not  to  explore  different  cost  engineering 
methodologies,  but  rather  to  discover  better  ways  of  performing  systems  engineering  to 
produce  more  meaningful,  relevant,  accurate,  high-confidence  information  that  serve  as 
inputs  to  the  models  and  estimate  analyses  of  cost  engineering;  all  with  a  focus  on  TOC. 

Design-to-Cost  (DTC)  and  Cost-as-an-Independent-Variable  (CAIV) 

The  Design-to-Cost  (DTC)  methodology  focuses  upon  projected  average  unit 
production  costs  (with  O&S  as  a  second-order  factor).  We  feel  the  DTC  process  and  metrics 
may  often  work  against  control  of  TOC.  With  the  emphasis  on  production  costs,  the 
program  management  team  may  obscure  the  long-term  TOC  issues  during  development  in 
order  to  satisfy  DTC  objectives. 
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Cost-as-an-Independent-Variable  (CAIV)  is  another  methodology  (like  DTC)  that 
uses  cost  as  an  end  goal.  In  CAIV,  cost  is  treated  as  an  independent  variable  among  the 
three  variables  traditionally  associated  with  a  defense  acquisition  program:  cost,  schedule, 
and  performance.  Cost  is  “fixed”  and  other  variables  traded  off  against  the  fixed  constraint  of 
cost.  Most  often,  the  “cost”  associated  with  CAIV  are,  indeed,  total  program  and  life  cycle 
costs  (i.e.,  TOC).  The  control  or  reduction  of  estimated  future  life-cycle  costs  are  considered 
as  important  as  trade-offs  to  meet  the  schedule  and  performance  thresholds  (Land,  1997). 

We  anticipate  that  CAIV  methodologies  are  very  closely  related  to  the  end  goals  of 
this  research.  Our  thesis  is  that  CAIV  methods  are  only  as  good  as  the  inputs  received  to 
perform  meaningful  cost  trade-offs.  The  goal  is  to  improve  those  inputs  and  processes 
related  to  SE  to  improve  upon  CAIV,  where  used.  Additionally,  we  are  searching  to  find 
opportunities  to  decompose  CAIV  into  other  components  such  as  RMA  (reliability, 
maintainability,  and  availability)  as  an  independent  Variable  (MAIV),  or  TOC  as  an 
independent  variable  (TAIV),  etc. 


NAVAIR  Technical  Approach 

The  technical  approach  for  this  research  is  shown  in  Figure  5.  The  desired  outcomes 
are  to  publish  best  practices  to  the  system  engineering  (primarily),  competency  engineering 
(e.g.,  aero  engineering,  mechanical  engineering,  etc.),  logistics,  and  program  management 
communities  at  NAVAIR.  We  also  seek  to  improve  existing  review  processes  (i.e.,  SE 
Technical  Reviews,  SETR).  Finally,  we  want  to  identify  and  be  able  to  assess  metrics  that 
emerge  from  system  engineering  that  can  provide  program  and  engineering  managers  an 
assessment  of  the  confidence  of  their  program’s  TOC  posture. 
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(Iterate) 


TOC  Tools 


Processes 


SETR 

Checklists 


Figure  5.  Technical  Approach  to  TOC  Research  at  NAVAIR 

As  stated  in  the  previous  section,  we  are  leveraging  well-established  methods  within 
the  DoD  and  in  industry  that  are  supported  by  recent  academic  research  (e.g.,  CAIV,  cost 
engineering,  etc.).  Using  that  as  a  baseline,  we  are  integrating  emerging  standards  related 
to  TOC  or  reliability,  availability,  and  maintainability  into  a  first-order  guidance  that  is  aligned 
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to  the  acquisition  cycle  milestones  and  gate  reviews.  Currently,  NAVAR  uses  a  checklist  tool 
that  aids  the  program  and  engineering  managers  to  navigate  the  technical  review  process 
(SETR),  and  we  are  evaluating  how  design  considerations  related  to  TOC  are  included  in 
those  lists  in  a  relevant  and  clear  manner,  as  well  as  considered  early  and  continually 
throughout  the  acquisition  process.  Finally,  we  are  examining  various  acquisition  program 
case  studies  to  understand  where  they  encountered  problems  in  TOC  estimation  or  control 
to  understand  best  practices  or  common  themes  that  could  reveal  necessary  remediation  in 
the  system  engineering  methods. 

Current  Findings 

Case  study  analysis,  interviews,  documentation  integration,  and  process 
improvement  activities  are  in  early  phases  and  have  yet  to  generate  major  discoveries.  This 
paper,  however,  lays  out  the  technical  approach  and  strategy  with  some  early  findings  that 
will  set  the  stage  for  continuing  dialogue  and  discussion  as  this  research  proceeds. 

Throughout  many  interviews  with  engineers  experienced  with  system  development  at 
NAVAIR,  certain  themes  are  emerging.  The  impacts  to  TOC  growth,  in  many  cases,  could 
be  categorized  as  caused  by  operational,  process,  and/or  design  issues  (see  Figure  6). 
Unanticipated  operational  tempo  or  harsh  real-world  environments  caused  TOC  growth  from 
the  exigencies  of  combat  operations  that  were  not  anticipated,  and  these  operations  were 
conducted  in  particularly  harsh  environments  (e.g.,  heat,  sand,  etc.).  Some  of  the  aircraft 
systems  were  of  unusually  high  complexity  and  introduced  new  technologies  unlike  previous 
aircraft.  This  dissimilarity  made  early  TOC  estimates  difficult  with  high  degrees  of 
uncertainty.  Finally,  processes  and  analyses  associated  with  reliability,  maintainability,  and 
availability  (RMA)  and  integrated  logistics  support  (ILS)  analyses  were  challenged  at  early 
design  phases;  that  also  resulted  in  high  variance  in  final  TOC.  These  three  dimensions 
appear  to  have  strong  mutual  coupling  of  their  dependencies  and  each  have  impacts  on 
TOC.  The  intent  is  to  discover  these  independencies  and  correlated  effects  through  case 
study  analysis. 
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Figure  6.  Key  Contributors  to  TOC  Derived  From  Case  Studies 


Documentation  and  Guidance 


ACQUISITION  RESEARCH:  CREATING  SYNERGY  FOR  INFORMED  CHANGE  -  72  - 


As  we  examine  the  emerging  guidelines  for  TOC  and  reliability,  maintainability  and 
availability,  we  found  through  interviews  and  user  interaction  that  the  documentation  had  not 
been  placed  into  common  usage.  It  became  apparent  that  many  documents  brought 
different  perspectives  that  were  difficult  for  the  engineering  teams  to  reconcile.  In  other 
cases,  the  documentation  did  not  translate  well  into  the  design  phase  or  acquisition  process 
of  the  program.  Additionally  there  was  little  quantitative  help  in  either  of  the  documents. 

We  were  able  to  parse  and  aggregate  the  emerging  standards  and  documentation 
where  appropriate  to  align  the  relevant  portions  of  each  to  the  proper  technical  reviews, 
major  milestones,  and  associated  gate  reviews.  Currently,  we  have  integrated  that 
information  into  a  tool  that  is  web-based  and  are  now  exposing  the  engineering  community 
to  the  tool  to  get  feedback  as  to  its  effectiveness. 

Figure  7  depicts  an  example  of  how  portions  of  documentation  are  being  aggregated 
to  align  with  the  major  milestones,  and  Figure  8  indicates  a  similar  alignment  of  the 
documentation  to  the  gate  reviews.  The  web-based  tool  allows  the  user  to  investigate  which 
TOC  issues  need  to  be  addressed  prior  to  the  reviews  or  the  milestones  as  they  progress  in 
the  development  cycle. 


0  (V)A  Q  Q  A 


'Document  the  development  of  the  RAM  Goals  (RAM-C  prZ.9j 

■  ID  maintenance/reliability  cost  drivers  &  Life-cycle  costs  (RAM-C  p.29) 

■  ID  risks  and  incorporate  into  initial  cost  estimate  4 RAM-C  p.2.9) 

■ID  sustainment  MQEs/MOPs  (RAM-C  p.2S) 

■  Document  lessons  learned  from  the  previously  used  systems  (RAM-C  p-23) 
■Develop  the  OMS/MP  (RAM-C  p,Z9) 


Figure  7.  Integrating  Standards  and  Aligning  to  Major  Acquisition  Milestones 
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•  Present  the  assumptions  and  elements  that  drive  your  TOC  estimate  (TOC  Guidebook  p,16| 

•  Present  life-cycle  perspective  of  the  distribution  of  TOC  by  appropriation  across  for  the  preferred  alternative  (include  comparisons  to  previous  and  TOC 
objectives}  (TOC  Guidebook  p.lG) 

•Present  an  overview  of  the  major  TOC  drivers  and  the  plans  to  mitigate  them  as  the  program  proceeds  (TOC  Guidebook  p,16j 


Figure  8.  Integrating  Standards  and  Aligning  to  Gate  Reviews 
Technical  Review  Checklist 

We  are  carefully  reviewing  the  large  body  of  SE  technical  review  items  that  are 
currently  used  in  the  NAVAIR  SETR  checklists  for  applicability  to  TOC.  We  are  exploring 
specific  checklist  items  that  pertain  to  TOC,  how  the  language  of  the  checklist  item  can  be 
strengthened  relative  to  TOC  objectives,  timing  of  the  items,  and  whether  they  expose  or 
stimulate  necessary  engineering  activity  to  help  increase  TOC  estimation  accuracy  at  the 
correct  phase  of  an  acquisition.  Figure  9  indicates  an  early  assessment  of  how  many  TOC 
related  checklist  items  are  addressed  at  each  design  review  as  prescribed  by  current 
checklist  policy.  Although  this  data  is  very  preliminary,  it  does  support  how  design  decisions 
related  to  TOC  are  most  appropriately  applied  early  in  the  program.  The  results  of  this 
activity  will  be  to  produce  a  set  of  refined  and  more  directed  checklist  items  (relative  to  TOC) 
they  can  be  reintegrated  into  the  existing  web-based  checklist  tool. 
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TOC/Readiness  Items  Relative  to  "Influence' 
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Figure  9.  TOC-Related  SETR  Checklist  Items  Appear  Appropriately  Front-Loaded 

During  Acquisition 


Reliability  Analysis 

Initial  case  study  interviews  indicate  there  is  a  strong  correlation  between  the  early 
reliability  and  maintainability  analysis  performed  on  the  program  and  the  quality  of  assessing 
the  TOC  of  the  program.  It’s  generally  accepted  at  the  working  level  that  reliability  analyses 
that  are  performed  during  the  early  design  phases  have  a  high  degree  of  uncertainty 
because  of  the  many  undefined  features  of  the  system  that  is  being  designed.  As  the  design 
proceeds,  reliability  analyses  become  more  accurate  as  they  reflect  more  and  more  actual 
components  that  will  comprise  the  system.  Unfortunately,  early  program  TOC  estimates 
must  be  based  on  these  early  reliability  analyses  which,  when  published,  can  overly  bias  the 
later  TOC  estimates  that  use  the  refined  RMA  data,  thus  creating  high  variance  in  total 
program  TOC  control.  We  will  continue  to  investigate  process  improvements,  additional 
metrics,  and  the  strength  of  correlation  between  TOC  growth  and  early  reliability 
estimations. 

Cost  Modeling 

Early  coordination  with  cost  estimation  organizations  confirms  that  while  those 
organizations  have  high  confidence  in  the  cost  models,  the  models  themselves  and  the 
resulting  outputs  are,  of  course,  dependent  on  the  quality  of  input.  Currently  we  are 
investigating  inputs  to  the  cost  model  related  to  the  technical  baseline  of  the  system,  spare 
parts,  depot  level  repair  strategy,  and  related  supply  chain  issues,  as  these  are  indicating  a 
strong  impact  on  model  performance. 

SE  Process  Alignment 

Together  with  aggregation  of  emerging  standards  into  a  guidance  document,  we  are 
also  investigating  how  to  align  SE  processes  to  the  TOC  objective  and  also  maintain 
alignment  with  technical  reviews  and  gate  reviews.  As  shown  in  Figure  10,  we  are  starting  to 
model  and  posit  alignment  of  SE  activities  that  will  be  explored  further  and  validated  within 
the  NAVAIR  community. 
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Material  Solution  Analysis  (MSA) 
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Figure  10.  Initial  SE  TOC-Related  Process  Alignment  to  Acquisition  Cycle 

Summary  and  Future  Research 

This  research  investigates  how  SE  processes  and  methods  can  be  improved  to 
increase  the  fidelity  of  TOC  estimation  and,  ultimately,  TOC  control  at  NAVAIR.  Early 
findings  are  beginning  to  point  to  the  conclusion  that  there  are  opportunities  for  improvement 
both  in  SE  processes  but  also  the  tools  and  documentation  and  guidance  which  is 
distributed  throughout  the  organization. 

We  are  investigating  existing  documentation  and  integrating  that  documentation  into 
guidelines  that  are  meaningful  to  TOC.  We  are  trying  to  enrich  existing  tools  (with  TOC 
relevance)  that  guide  engineering  and  acquisition  teams  through  the  technical  development 
and  review  process.  We  are  investigating  historical  and  ongoing  aircraft  acquisition 
programs  to  discover  best  practices  and  to  reveal  opportunities  for  processes  to  improve 
TOC.  Finally,  we  are  in  the  early  phases  of  uncovering  SE  metrics  that  may  aid  in  the 
development  of  tools  for  the  program  management  and  engineering  management  teams  to 
assess  the  TOC  posture  of  their  program.  Further  research  will  continue  along  the  activities 
outlined  in  this  paper  and  will  be  shared  in  subsequent  papers  and  forms. 
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Goals  and  Objectives 


•  Problem:  Systems  engineering  (SE)  is  optimized  for 
designing  for  best  system  performance  but  may  not  be 
aligned,  prioritized,  or  defined  well  to  designing  for 
affordability  and  TOC  objectives. 

-  Research  question  1:  How  does  SE  contribute  to  TOC 
estimation,  reduction,  and  control  objectives  and  activities? 

-  Research  question  2:  Can  SE  activities  be  improved,  better 
defined,  or  integrated  into  other  TOC  reduction  activities  to 
improve  TOC  estimation  and  control? 

-  Research  question  3:  Can  TOC  metrics  be  developed  and 
integrated  into  SE  and  program  activities  and  toolsets  to 
quantitatively  develop  TOC  KPPs,  KSAs,  MOEs,  etc.  and 
quantitatively  assess  program  performance  against  those  metrics? 
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What  is  TOC? 


TOC  is  comprised  of  costs  to  research,  develop,  acquire, 
own,  operate,  and  dispose  of  weapon  and  support  systems, 
other  equipment  and  real  property,  the  costs  to  recruit,  train, 
retain,  separate  and  otherwise  support  military  and  civilian 
personnel,  and  all  other  costs  of  business  operations  of  the 
DoD”.  (Gansler.  1998V 

Research  and  development  cost  which  extend  from  the  concept 
phase  to  the  technology  development  phase  and  through  to 
development  and  demonstration. 

Costs  associated  with  system  production 
Operations  and  support  cost  during  sustainment  phase 
Disposal  and  retirement  costs 


Components  of  TOC  (NAVAIR) 


Total  Ownership  Cost 

■+■  Common  Spares/  Support  items.  Infrastructure  Cost  for  Planning/  Managing/  Operating/  Executing,  Linked 
Indirect  Costs,  Modification  Improvements 


Life  Cycle  Cost 

■*  Operations  And  Support  ( Mission  Personnel.  Unit-level  Consumption,  Intermediate  Maintenance, 
Depot  Maintenance,  Contractor  Support.  Sustaining  Support,  indirect  Support;,  Disposal 

Acquisition  Cost 

*  RDT&E,  Facility  Construction 


Procurement  Cost 

*  Initial  Spares 


Weapon  System  Cost 

+  Tech  Data,  Publications,  Contractor  Services,  Support  Equipment, 
Training  Equipment,  Factory  Training 

Flyaway  Cost 

+  Non-recurring,  Ancillary  Equipment 

Recurring  Flyaway 

*  Management.  Hardware,  ECOs 
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Afloat  Life  Cycle  Costs 


The  TOC  Problem 


[* - System  Acquisition - *| 
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Operation  & 
Support 


TOC  growth 


Development  Milestones 


Actual  cost  growth 
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Previous  Related  Methodologies 


•  Value  MethodOlOQy  /  Value  I_I  iyn  icci  II  iy 

•  Cost  Engineering 

•  Design  to  Cost  (DTC) 

•  Cost  as  an  independent  variable  (CAIV) 
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SE  technical  review 
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Existing  TOC  Documentation  / 
Guidance 

•  Lack  of  common  usage 

•  Different  perspectives 

•  Did  not  translate  well  into  the  design  pnase 

•  Little  quantitative  prescriptions 
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TOC  Guidance  Aligned  to  Milestones 
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•Document  the  development  of  the  RAM  Goals  (RAM*C  p,29) 

•ID  maintenance/reliability  cost  drivers  &  Life-cycle  costs  (RAM-C  p.29) 

•ID  risks  and  incorporate  into  initial  cost  estimate  (RAM-C  p-29) 

•ID  sustainment  MOEs/MOPs  (RAM-C  p*29) 

•  Document  lessons  learned  from  the  previously  used  systems  (RAM-C  p.29) 
•Develop  the  OMS/MP  (RAM-C  p.29) 
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TOC  Guidance  Al. _ to  Gates 
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POR  PDR II  IBR 

•  Present  the  current  and  past  SCP  or  SYSCOM  estimate  to  the 
APB  threshold  {TOC  Guidebook  p+20) 

•  Present  a  Logistics  Requirements  and  Funding  Summary 
(IRFSJ  {TOC  Guidebook  p<20) 

•  Present  a  historical  perspective  of  TOC  estimate  changes 
over  ali  previous  gates  and  milestones  (TOC  Guidebook  p*20) 


•Present  the  assumptions  and  elements  that  drive  your  TOC  estimate  (TOC  Guidebook  p .16) 

•Present  life-cycle  perspective  of  the  distribution  of  TOC  by  appropriation  across  for  the  preferred  alternative  (include  comparisons  to  previous  and  TOC 
objectives)  (TOC  Guidebook  p*lG) 

•Present  an  overview  of  the  major  TOC  drivers  and  the  plans  to  mitigate  them  as  the  program  proceeds  (TOC  Guidebook  p.16) 
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SET. .  Checklists  (T0_  emphasis) 


•  SETR  checklists  specify 
questions  that  prepare 
engineering  teams  for 
reviews 

•  SETR  checklist  tool  is  web- 
based 

•  Checklist  questions  are 
being  strengthened  to 
reflect  TOC  objectives  and 
timing  (based  upon 
lessons-learned) 
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Reliability  Analysis 


•  Strong  correlation  between  early  reliability 
analysis  and  accuracy  of  TOC  projections 

•  “Catch-22” 


-  Early  RAM  estimates  needed  for  TOC  estimates 

-  Earl  RAM  is  based  u  on  immature  desi  n 

■s  ■  w 

-  Early  TOC  estimates  can  become  “anchored” 


•  Investi  ation  continues  for  rocess 
improvements,  metrics,  and  quantifying 
above  correlation  strength 
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Cost  Modeling 

•  Cost  Modeling 

-  Cost  models  are  mature  and  trusted 

-  “TOC”  inputs  (particular.,  t _ ri _ _ _ 

RAM  analysis)  may  have  high  variance  and 
uncertainty 

-  Results  may  have  high  TOC  estimate  variance 
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SE  Process  Alignment 


•  How  can  SE  processes  be  aligned,  more  integrated, 
or  optimized  to  reflect  higher  accuracy  or  TOC 


estimates  early? 

Early  SE 
process 
alignment 
concepts:  — > 


Material  Solution  Analysis  (MSA) 


i  System 
A  Requirement 

A 

Technology 

Development 


System 
Architecture  & 
Technology 
3emonst  ratio! 


1 


Engineering  & 
Manufacturing 
Development 


■TOC  A&sumpOafH 


TOC- Wig 
'  Risks  ' 


TOC-U*saem  Pack** 


"AfTorelBbikl’j1 


▼ 

TOC  AM^npIkJn* 

TOC  Tqj>4s¥b!  BstFnalsa 
Mission-TOC  cos!  mpacts 
OWS'MP 

TOCMOEa/MOPs 


T 

TOC  TPMs 
TOC  lacm  n  SPS 
TOC  SEP  inputs 

Pfrcsduqt  Suppcxt  Strategy  Plan  (draft  |i 


▼ 

TOC  history 
LflfS  I LRSP 
I  LA 

Prodiic:  Support  Straiagy  Plan  (Inal) 
SEPrTEMP  (updtKSli 


Acquisition  EesKureh  Program;  Creating  Synergy  for  Informed  Change 


Nava  I  rs^iRiMilLtaic  School 

Monlcrejp  CA 


i 

22 


Summary 


•  Goal:  How  can  SE  processes  and  methods 
can  be  improved  to  increase  the  fidelity  of 
TOC  estimation  and,  ultimately,  TOC  control 
at  NAVAIR? 

•  Early  findings:  oemmm, 

+  w  •  Tempo 

-  Existing  documentation  and  guidance 

-  SE  technical  review  checklists 


-  Reliability  analysis 

-  Cost  modeling 

-  SE  process  alignment 
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